TITLE OF THE INVENTION 
COLOR BALANCE ADJUSTMENT OF IMAGE SENSED UPON EMITTING 

FLASH LIGHT 

5 FIELD OF THE INVENTION 

The present invention relates to an image sensing 
apparatus for sensing, recording, and reproducing a 
still image and moving image, and an image sensing 
apparatus control method. 

10 

BACKGROUND OF THE INVENTION 

Image sensing apparatuses such as digital cameras 
and the like, which record and reproduce still images 
and moving Images using a memory card having a 

15 solid-state memory element as a recording medium are 
already commercially available, and digital cameras 
having an auto white balance function are also 
commercially available. 

With these digital cameras, a digital camera user 

20 can take an image regardless of the color temperature 
of a light source with which an object is irradiated. 
When a flash light is used to illuminate an object upon 
sensing an image, there are two different light sources, 
i.e., external light and flash light upon emitting 

25 flash light. At this time, the color temperatures of 
external light and flash light are independently 
detected. Then, white balance is determined from the 
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influences of the color temperatures of external light 
and flash light based on the ratio of irradiation 
luminance. 

Such conventional image sensing apparatus such as 
5 a digital camera or the like, holds a fixed value of 
the color temperature of the flash light as a designed 
value or stores a color temperature obtained by 
emitting the flash light upon individual adjustment of 
the image sensing apparatus at the time of production 

10 in a factory. 

However, the color temperature of the flash light 
changes depending upon an emission amount, and repeated 
emission causes variation in the color temperature. As 
a result, the actual color temperature of the flash 

15 light varies for each emission of the flash light. If 
the actual color temperature largely differs from the 
stored fixed value or individual adjustment value, the 
color balance of a sensed image does not coincide with 
the actual color temperature. 

20 Further, when the distance between the camera and 

object is long, since importance is attached to the 
color temperature of flash light even when the ratio of 
external light is large, a sensed image suffers 
inappropriate color balance. 

25 

SUMMARY OF THE INVENTION 
The present invention has been made in 
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consideration of the above situation, and has as its 
object to attain appropriate color balance correction 
of a sensed image when an image sensing is made while 
emitting flash light. 
5 According to the present invention, the foregoing 

object is attained by providing a method of controlling 
an image sensing apparatus, which has an image sensor, 
a flash light used to illuminate an object with light, 
and an image sensing instruction unit, comprising: 

10 sensing a first image while causing the flash light to 
emit light in response to an image sensing designation 
by the image sensing instruction unit; calculating a 
color temperature for processing the first image using 
at least color temperature information of the first 

1 5 image . 

According to the present invention, the foregoing 
object is also attained by providing a controller for 
an image sensing apparatus, which has an image sensor, 
a flash light used to illuminate an object with light, 

20 and an image sensing instruction unit, comprising: a 
first image sensing controller which controls to sense 
a first image while causing the flash light to emit 
light in response to an image sensing designation by 
the image sensing instruction unit; a color temperature 

25 calculation unit which calculates a color temperature 
for processing the first image using at least color 
temperature information of the first image. 
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Further, the foregoing object is also attained by 
providing an image sensing apparatus including the 
above controller. 

Other features and advantages of the present 
5 invention will be apparent from the following 

description taken in conjunction with the accompanying 
drawings, in which like reference characters designate 
the same or similar parts throughout the figures 
thereof . 

10 

BRIEF DESCRIPTION OF THE DRAWINGS 
The accompanying drawings, which are incorporated 
in and constitute a part of the specification, 
illustrate an embodiment of the invention and, together 
15 with the description, serve to explain the principles 
of the invention. 

Fig. 1 is a block diagram showing the arrangement 
of an image sensing apparatus according to an 
embodiment of the present invention; 
20 Fig. 2 is a flow chart showing a part of a main 

routine according to the embodiment of the present 
invention; 

Fig. 3 is a flow chart showing a part of the main 
routine according to the embodiment of the present 
25 invention; 

Fig. 4 is a flow chart of a distance 
measurement /photometry routine according to the 
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embodiment of the present invention; 

Fig. 5 is a flow chart of an image sensing 
routine according to the embodiment of the present 
invention; 

Fig. 6 is a flow chart of a recording routine 
according to the embodiment of the present invention; 

Fig. 7 is a flow chart of a white balance process 
routine according to the embodiment of the present 
invention ; and 

Fig. 8 is a timing chart of an image sensing 
sequence according to the embodiment of the present 
invention . 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 
15 A preferred embodiment of the present invention 

will now be described in detail in accordance with the 
accompanying drawings . 

Fig. 1 is a block diagram showing the arrangement 
of an image sensing apparatus with an image processing 
20 function according to an embodiment of the present 
invention . 

Referring to Fig. 1, reference numeral 100 
denotes an image sensing apparatus . Reference numeral 
10 denotes an image sensing lens; 12, a shutter having 
25 a diaphragm function; 14, an image sensing device which 
converts an optical image into an electrical signal; 
and 16, an A/D converter which converts an analog 
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signal output from the image sensing device 14 into a 
digital signal. 

Reference numeral 18 denotes a timing generator 
which supplies a clock signal and control signal 
5 respectively to the A/D converter 16 and a D/A 

converter 26 under the control of a memory controller 
22 and system controller 50. 

Reference numeral 20 denotes an image processor 
which executes a predetermined pixel interpolation 

10 process and color conversion process for data output 

from the A/D converter 16 or memory controller 22. The 
image processor 20 executes a predetermined calculation 
process using image data output from the A/D converter 
16. Based on the obtained calculation result, the 

15 system controller 50 executes a through- the -lens (TTL) 
auto focus (AF) process, auto exposure (AE) process, 
and pre- flash (EF) process with respect to an exposure 
controller 40 and distance measurement controller 42. 
Furthermore, the image processor 20 executes a 

20 predetermined calculation process using image data 

output from the A/D converter 16, and also executes a 
TTL auto white balance process (AWB) on the basis of 
the obtained calculation result. 

The memory controller 22 controls the A/D 

25 converter 16, the timing generator 18, the image 
processor 20, an image display memory 24, the D/A 
converter 26. a memory 30, and a compression/expansion 
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circuit 32. Image data output from the A/D converter 
16 is written in the image display memory 24 or memory 
30 via the image processor 20 and memory controller 22 
or via the memory controller 22 alone - 
5 Reference n\imeral 24 denotes the image display 

memory; 26, the D/A converter; and 28, an image display 
unit which comprises a TFT LCD or the like. Image data 
to be displayed written in the image display memory 24 
is displayed on the image display unit 28 via the D/A 

10 converter 26. An electronic viewfinder function can be 
realized by sequentially displaying sensed image data 
using the image display unit 28. The image display 
unit 28 arbitrarily turns on/off its display in 
accordance with an instruction from the system 

15 controller 50. If the display is turned off, the power 
consumption of the image sensing apparatus 100 can be 
greatly reduced. 

The memory 30 is used to store sensed still 
images and moving images. The memory 30 has a storage 

20 capacity which is large enough to store a predetermined 
number of still images and a moving image/ images for a 
predetermined period. In a sequential -shot image 
sensing mode for sequentially sensing a plurality of 
still images or a panoramic image sensing mode, 

25 large- size image data can be written in the memory 30 
at a high speed. 

The memory 30 can also be used as a work area of 
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the system controller 50. 

The compression/expansion circuit 32 compresses 
or expands image data by adaptive discrete cosine 
transformation (ADCT) or the like. The 
5 compression/expansion circuit 32 reads image data 
stored in the memory 30, executes a compression or 
expansion process on the read image data, and writes 
the processed data in the memory 30 again. 

The exposure controller 40 controls the shutter 

10 12 having the diaphragm function. The exposure 

controller 40 also has a flash adjusting function in 
collaboration with a flash 48. The distance 
measurement controller 42 controls focusing of the 
image sensing lens 10. Reference numeral 44 denotes a 

15 zoom controller which controls zooming of the image 
sensing lens 10; and 46, a barrier controller which 
controls the operation of a barrier 102. The flash 
(flash device) 48 also has an AF auxiliary light 
projection function, and a flash adjusting function 

20 used to illuminate an object with light. The system 
controller 50 controls the exposure controller 40 and 
distance measurement controller 42 by the TTL method on 
the basis of the result of calculations by the image 
processor 20 using image data output from the A/D 

25 converter 16. 

The system controller 50 controls the overall 
image sensing apparatus 100. Reference numeral 52 
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denotes a memory which stores constants, variables, 
programs, and the like required to operate the system 
controller 50. 

Reference numeral 54 denotes a notification unit 
5 which notifies an external apparatus of operation 

status, messages, and the like using characters, images, 
sound, and the like in accordance with the execution 
state of a program in the system controller 50 . For 
example, the notification unit 54 comprises a 

10 combination of one or more devices of display devices 
including an LCD and LEDs for visual notification, and 
sound generating devices for audible notification. 
Especially, the display device or devices is /are 
provided at one or a plurality of visually recognizable 

15 positions near around an operation unit 70. Further, 

some functions of the notification unit 54 are provided 
within an optical viewfinder 104. 

Of the display contents of the notification unit 
54 , those to be displayed on the LCD or the like 

20 include indications of single-/sequential-shot image 
sensing, a self timer, a compression rate, the number 
of recordable pixels, the number of recorded images, 
the number of recordable images, a shutter speed, an 
f -number (aperture value), exposure compensation, flash 

25 illumination, pink-eye effect mitigation, macro image 
sensing, a buzzer-set state, a timer battery level, a 
battery level, an error state, information using plural 
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digit niombers, attached/ detached state of recording 
media 200 and 210, operation of a communication I/F, 
and date and time. 

Of the display contents of the notification unit 
5 54, those to be displayed within the optical viewfinder 
104 include indications of an in-focus state, camera 
shake warning, a flash charge state, a shutter speed, 
an f -number, exposure compensation, and the like. 

Reference numeral 56 denotes an electrically 
10 erasable and recordable nonvolatile memory such as an 
EEPROM or the like. 

Reference numerals 60, 62, 64, and 70 denote 
operation means for inputting various operation 
instructions to the system controller 50. These 
15 operation means comprise a combination of one or a 

plurality of switches, dials, a touch panel, a pointing 
device by means of line-of -sight detection, a speech 
recognition device, and the like. 

Next, these operation means will be described in 
20 more detail below. 

Reference numeral 60 denotes a mode dial switch 
for switching various function modes such as a power 
OFF mode, automatic image sensing mode, program image 
sensing mode, panoramic image sensing mode, 
25 reproduction mode, multi-image reproduction/deletion 
mode, PC connection mode, and the like. 

Reference numeral 62 denotes a shutter switch SWl, 
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which Is turned on upon depressing a shutter button 
(not shown) to its half- stroke position, and instructs 
to start operations such as an AF process, AE process, 
AWB process, EF process, and the like. 
5 Reference numeral 64 denotes a shutter switch SW2, 

which is turned on upon depressing the shutter button 
(not shown) to its full- stroke position, and instructs 
to start a series of processes including an exposure 
process for writing a signal read out from the image 

10 sensing device 14 in the memory 30 as image data via 
the A/D converter 16 and memory controller 22, a 
development process using calculations in the image 
processor 20 and memory controller 22, and a recording 
process for reading out image data from the memory 30 , 

15 compressing the readout image data by the 

compression/expansion circuit 32, and writing the 
compressed image data in the recording medium 200 or 
210. 

Reference numeral 70 denotes an operation unit 
20 which comprises various buttons and a touch panel. The 
operation unit 70 includes a menu button, set button, 
macro button, multi- image reproduction /repaging button, 
flash set button, 

single-shot/sequential-shot/self -timer select button, 
25 forward (+) menu item select button, backward (-) menu 
item select button, forward (+) reproduction image 
search button, backward (-) reproduction image search 
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button, image sensing quality select button, exposure 
correction button, date/time set button, and the like. 

Reference numeral 80 denotes a power controller 
which comprises a battery detection circuit, DC-DC 
5 converter, a switch circuit used to select a block to 
be energized, and the like. The power controller 80 
detects the attached/detached state of the battery, 
battery type, and remaining batteiry power level, 
controls the DC-DC converter on the basis of the 

10 detection results and an instruction from the system 
controller 50, and supplies required voltages to 
respective units including the recording medium for a 
required period of time. 

Reference numerals 82 and 84 denote connectors; 

15 and 86, a power source which comprises a primary 

battery such as an alkaline battery, lithium battery, 
or the like, a secondary battery such as an NiCd 
battery, NiMh battery, Li-ion battery, or the like, an 
AC adapter, and the like. 

20 Reference numerals 90 and 94 denote interfaces 

with recording media such as a memory card, hard disk, 
and the like; 92 and 96, connectors for connecting with 
the recording media such as a memory card, hard disk, 
and the like; and 98, a recording medium 

25 attached/detached state detector for detecting whether 
or not the recording mediiim 200 and/or the recording 
medium 210 are/is attached to the connector 92 and/or 
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the connector 96. The recording medium 
attached/detached state detector 98 can also detect 
whether or not various communication cards and the like 
(to be described later) are attached. In addition to 
5 the recording medium 200 or 210. 

In the above description, this embodiment employs 
two systems of Interfaces and connectors for connecting 
recording media. However, the number of systems is not 
limited, and one or a plurality of interfaces and 

10 connectors for connecting recording media may be 

employed. Further, Interfaces and connectors pursuant 
to different standards may be combined. 

As the Interfaces and connectors, those which 
comply with the PCMCIA (Personal Computer Memory Card 

15 International Association) card standards, CF (Compact 
Flash) card standards, and the like may be used. 

Furthermore, when interfaces and connectors, 
which comply with the PCMCIA card standards , CF card 
standards, or the like, are used as the interfaces 90 

20 and 94 and the connectors 92 and 96, image data and 

management information attached to the image data can 
be exchanged with other computers and peripheral 
devices such as a printer and the like by connecting 
various communication cards such as a LAN card, modem 

25 card, USB card, IEEE1394 card, P1284 card, SCSI card, 
PHS card, and the like. 

The barrier 102 serves as a protection device. 
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which covers the image sensing portion including the 
lens 10 of the image sensing apparatus 100 to protect 
it from being contaminated or damaged. 

The optical viewfinder 104 can be used to sense 
5 an image without using an electronic viewfinder 

function provided by the image display unit 28. Within 
the optical viewfinder 104, some functions of the 
notification unit 54, e.g., an in-focus state, camera 
shake warning, flash charge, shutter speed, f- number, 

10 exposure correction, and the like are displayed. 

Reference numeral 110 denotes a communication 
circuit which has various communication functions such 
as RS232C, USB, IEEE1394, P1284, SCSI, modem, LAN, 
wireless communications, and the like. 

15 Reference numeral 112 denotes a connector or 

antenna (in case of a wireless communication) for 
connecting the image sensing apparatus 100 with another 
apparatus via the communication circuit 110. 

The recording medium 200 comprises a memory card, 

20 hard disk, or the like. The recording medium 200 has a 
recording unit 202 which comprises a semiconductor 
memory, magnetic disk, or the like, an interface 204 
with the image sensing apparatus 100, and a connector 
206 for connecting with the image sensing apparatus 100. 

25 Also, the recording medium 210 comprises a memory card, 
hard disk, or the like. The recording medium 210 has a 
recording unit 212 which comprises a semiconductor 
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memory, magnetic disk, or the like, an interface 214 
with the image sensing apparatus 100, and a connector 
216 for connecting with the image sensing apparatus 100. 
The operation of the image sensing apparatus 100 
5 with the above arrangement according to the embodiment 
of the present invention will be described below with 
reference to Figs. 2 to 8. 

Figs. 2 and 3 are flow charts of the main routine 
of the image sensing apparatus 100 of this embodiment. 

10 Referring to Fig. 2, upon power ON after, e.g., 

battery exchange, the system controller 50 initializes 
flags, control variables, and the like (step SlOl), and 
disables image display of the image display unit 28 as 
an initial state (step S102). In step S103. the system 

15 controller 50 checks the setting position of the mode 
dial 60. If the mode dial 60 is set at the power OFF 
mode position, the system controller 50 executes a 
predetermined end process (step S105). In this process, 
the system controller 50 changes display of the 

20 respective display units to an end state, closes the 
barrier 102 to protect the image sensing portion, 
records necessary pareimeters including flags, control 
variables, and the like, setup values, and setup mode 
in the nonvolatile memory 56, and controls the power 

25 controller 80 to shut off unnecessary power supplies to 
the respective units of the image sensing apparatus 100 
including the image display unit 28. After that, the 
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flow returns to step S103. 

If it is determined in step S103 that the mode 
dial 60 is set at the image sensing mode position, the 
flow advances to step S106. 
5 On the other hand, if it is determined in step 

S103 that the mode dial 60 is set at another mode 
position, the system controller 50 executes a process 
corresponding to the selected mode (step S104). Upon 
completion of the process, the flow returns to step 

10 S103. The miscellaneous modes in step S104 include, 

e.g., an image confirmation mode. In this mode, index 
images are displayed to confirm sensed images, and 
modification, retouching, filing, and the like of 
acquired images can be done. 

15 The system controller 50 determines using the 

power controller 80 in step S106 whether or not the 
remaining level and operation status of the power 
source 86 cause problems in the operation of the image 
sensing apparatus 100. If any problems are found (NO 

20 in step S106), a predetermined warning message is 
visually and/or audibly generated using the 
notification unit 54 (step S108). The flow then 
returns to step S103. 

On the other hand, if the power source 86 has no 

25 problem (YES in step S106), the system controller 50 
determines in step S107 whether or not the operation 
status of the recording medium 200 or 210 causes 
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problems in the operation of the image sensing 
apparatus 100, especially, the recording/reproduction 
operation of image data to/from the recording medium 
200 or 210. If any problems are found (NO in step 
5 S107), a predetermined warning message is visually 

and/or audibly generated using the notification unit 54 
(step S108). The flow then returns to step S103. 

If the operation status of the recording medium 
200 or 210 has no problem (YES in step S107), the 

10 system controller 50 visually and/or audibly notifies 

various setup states of the image sensing apparatus 100 
using the notification unit 54 (step S109). Note that 
the system controller 50 may enable image display of 
the image display unit 28, and visually and audibly 

15 notify various setup states of the image sensing 
apparatus 100 using the image display unit 28. 

In step S116, the system controller 50 enables 
image display of the image display unit 28 and sets it 
in a through display mode to sequentially display 

20 sensed image data. The flow then advances to step S119 
in Fig. 3. In the through display mode, data, which is 
sequentially written in the image display memory 24 via 
the image sensing device 14, A/D converter 16, image 
processor 20, and memory controller 22, is sequentially 

25 displayed on the image display unit 28 via the memory 
controller 22 and D/A converter 26, thus implementing 
an image monitor function. 
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If it is determined in step S119 that the shutter 
switch SWl is OFF, the flow returns to step S103. If 
the shutter switch SWl is ON, the flow advances to step 
S122. The system controller 50 executes a distance 
5 measurement process to adjust the focal point of the 
image sensing lens 10 to an object, and also a 
photometry process to determine an aperture value and 
shutter speed (step S122). As a result of photometry, 
the system controller 50 sets a flash flag if necessary 

10 to use the flash. Details of the distance 

measurement /photometry process in step S122 will be 
described later with reference to Fig. 4. 

Upon completion of the distance 
measurement /photometry process (step S122), the flow 

15 advances to step S125. If the shutter switch SW2 is 

OFF and the shutter switch SWl is canceled (step S126), 
the flow returns to step S103. If the shutter switch 
SW2 is ON (step S125), the system controller 50 checks 
if the flash flag is set as a result of the photometry 

20 process in step S122 (step S127). If the flash flag is 
set, i.e., if an image must be sensed while emitting 
flash light, the system controller 50 records a through 
display image as an immediately preceding image in step 
S128. 

25 In the present invention, the immediately 

preceding image recorded in this step is used in a 
white balance process executed in step S402 (will be 
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described later). By obtaining the color temperature 
of external light and the ratio of brightness between 
the external light and flash light from this 
immediately preceding image, appropriate color balance 
of an image sensed upon emitting flash light can be 
obtained. Note that the brightness can be, e.g., light 
amounts, light levels (Y), AE values, or combination 
thereof. The processes from capture of this 
immediately preceding image until an actual image 
sensing process will be described in detail later with 
reference to the timing chart of Fig. 8. 

On the other hand, if an image is to be sensed 
without emitting flash light, since no process using 
the immediately preceding image is required, the flow 
jumps from step S127 to step S129. 

In step S129, the system controller 50 executes 
an image sensing process that includes an exposure 
process for writing sensed image data in the image data 
memory 30 via the image sensing device 14, A/D 
converter 16, image processor 20, and memory controller 
22 or directly from the A/D converter 16 via the memory 
controller 22, and a development process for reading 
out image data written in the image data memory 30 
using the memory controller 22 and image processor 30 
as needed, and executing various processes. Details of 
the image sensing process in step S129 will be 
described later using Fig. 5. 



upon completion of the Image sensing process, the 
flow advances to step S134, and the system controller 
50 executes a recording process (step S134). In this 
process, the system controller 50 reads out sensed 
5 image data written in the image data memory 30, and 
executes various image processes for the readout image 
data using the memory controller 22 and image processor 
20 as needed. Also, the system controller 50 executes 
an image compression process according to the set mode 
10 using the compress ion /expansion circuit 32. After that, 
the system controller writes the image data on the 
recording medium 200 or 210. Details of the recording 
process in step S134 will be described later using 
Fig. 6. 

15 Upon completion of the recording process (step 

S134), if the shutter switch SW2 is kept ON (step S135), 
the system controller 50 checks a sequential- shot flag 
stored in its internal memory or the memory 52 (step 
S136). If the sequential-shot flag is set, the flow 

20 returns to step S129 to sequentially execute the image 
sensing process, thus sensing the next image. 

On the other hand, if the sequential-shot flag is 
not set (NO in step S136), the flow returns to step 
S135 to repeat the current process until the shutter 

25 switch SW2 is released. 

If the shutter switch SW2 is released, the state 
of the switch SWl is checked in step S141. If the 
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shutter switch SWl is ON, the flow returns to step S125, 
and the system controller 50 prepares for the next 
image sensing process. On the other hand, if the 
shutter switch SWl is OFF, the flow returns to step 
5 S103, and the system controller 50 ends a series of 
image sensing operations . 

Fig. 4 is a detailed flow chart of the distance 
measurement /photometiry process in step S122 in Fig. 3. 
The system controller 50 reads out a charge signal from 

10 the image sensing device 14, and sequentially inputs 

sensed image data to the image processor 20 via the A/D 
converter 16 (step S201). Using this sequentially read 
image data, the image processor 20 executes 
predetermined calculations used in the TTL AE process, 

15 EF process, and AF process. 

Note that these processes extract a required 
number of specific portions from all sensed pixels as 
needed and use them in calculations. In this way, in 
the TTL AE, EF, AWB , and AF processes, optimal 

20 calculations can be.made for respective different modes 
such as a center emphasis mode, average mode, and 
evaluation mode. 

The system controller 50 executes AE control 
using the exposure controller 40 using the calculation 

25 results of the image processor 20 (step S203) until an 
appropriate exposure (AE) value is obtained (until YES 
is determined in step S202). The flow then advances to 
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step S204 , and the system controller 50 checks using 
measurement data obtained by the AE control if flash 
light is required (step S204). If flash light is 
required (YES in step S204), the system controller 50 
5 sets a flash flag, and charges the flash 48 (step S205). 

If it is determined that an appropriate exposure 
(AE) value is obtained (YES in step S202), the 
measurement data and/or setup parameters are stored in 
the internal memory of the system controller 50 or the 

10 memory 52 . The system controller 50 then executes a 

distance measurement process (AF) using the calculation 
results of the image processor 20 and the measurement 
data obtained by the AE control, and repeats AF control 
using the distance measurement controller 42 (step 

15 S209) until an in-focus point is detected (while NO is 
determined in step S208). 

If it is determined that an in-focus point is 
detected as a result of distance measurement (AF) (YES 
in step S208), the measurement data and/or setup 

20 parameters are stored in the internal memory of the 

system controller 50 or the memory 52, thus ending the 
distance measurement /photometry process routine (step 
S122) . 

Fig. 5 is a detailed flow chart of the image 
25 sensing process in step S129 in Fig. 3. 

The system controller 50 determines in step S320 
whether or not the flash 48 is required to emit light 
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by checking if the flash flag is set. If no flash 
light is required (NO in step S320) , the flow jumps to 
step S301; otherwise (YES in step S320). the system 
controller 50 controls the exposure controller 40 on 
the basis of the photometry data stored in its internal 
memory or the memory 52 to open the shutter 12 having 
the diaphragm function in accordance with the aperture 
value, so as to expose the image sensing device 14 
(step S321), and to measure an external light luminance 
level without flash emission (step S322). Subsequently, 
pre-emission is made, and a luminance level upon 
emitting flash light is measured (step S323). The 
emission amount is calculated from the obtained two 
photometry data, and is stored in the internal memory 
of the system controller 50 or the memory 52. The 
control then enters an actual image sensing process. 

The system controller 50 controls the exposure 
controller 40 in accordance with the photometry data 
stored in its internal memory or the memory 52 to set 
the shutter 12 having the diaphragm function to an 
aperture value corresponding to the obtained photometry 
data, thus exposing the image sensing device 14 (steps 
S301 and S302) . 

The system controller 50 determines in step S303 
whether or not the flash 48 is required by checking if 
the flash flag is set. If the flash 48 is required, 
the system controller 50 controls the flash 48 to emit 



light in accordance with the flash emission amount 
stored in its internal memory or the memory 52 in step 
S323 (step S304). The system controller 50 waits for 
the end of exposure of the image sensing device 14 in 
5 accordance with the photometry data (step S305), and 
closes the shutter 12 (step S306). Then, the system 
controller 50 reads out a charge signal from the image 
sensing device 14, and writes sensed image data in the 
image data memory 30 via the A/D converter 16, image 
10 processor 20, and memory controller 22, or directly 

from the A/D converter 16 via the memory controller 22 
(step S307) . 

If a frame process is required in accordance with 
the set image sensing mode (YES in step S308), the 

15 system controller 50 reads out image data written in 
the image data memory 30 via the memory controller 22 
and image processor 20 as needed, and sequentially 
executes a vertical addition process (step S309) and 
color process (step S310). After that, the system 

20 controller 50 writes the processed image data in the 
image data memory 30. 

The system controller 50 reads out image data 
from the image data memory 30, and transfers display 
image data to the image display memory 24 via the 

25 memory controller 22 (step S311). Upon completion of a 
series of processes, the system controller 50 ends the 
image sensing process routine (step S129). 
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Fig. 6 is a detailed flow chart of the recording 
process in step S134 in Fig. 3. The system controller 
50 reads out sensed image data written in the image 
data memory 30 using the memory controller 22 and image 
processor 20 as needed, and executes a pixel square 
formation process for interpolating the aspect ratio of 
the image sensing device to 1 : 1 (step S401). The 
system controller 50 then writes the processed image 
data in the image data memory 30. 

In step S402, the system controller 50 reads out 
image data written in the image data memory 30 and 
executes a white balance process. Details of the white 
balance process executed in step S402 will be described 
later with reference to Fig. 7. 

The system controller 50 then reads out image 
data written in the image data memory 30 and executes 
an image compression process according to the set mode 
using the compression/expansion circuit 32 (step S403). 
The system controller 50 writes the compressed image 
data in the recording medium 200 or 210 such as a 
memory card, compact flash® card, or the like via the 
interface 90 or 94 and the connector 92 or 96 (step 
S404). Upon completion of the write process in the 
recording medium, the system controller 50 ends the 
recording process routine (step S134). 

Fig. 7 is a detailed flow chart of the white 
balance process in step S402 in Fig. 6. 



The system controller 50 checks if flash light 
has been emitted (step S701). If no flash light has 
been emitted, the system controller 50 detects the 
color temperature of external light from the sensed 
5 image (step S702), and executes a known white balance 
process according to the detected color temperature in 
step S707. On the other hand, if flash light has been 
emitted, the system controller 50 detects the color 
temperature of external light from the immediately 

10 preceding image before image sensing which is recorded 
in step S128 in Fig. 3 (step S703), and detects the 
color temperature of flash light from the image 
(actually sensed image) obtained by the actual image 
sensing process in step S129 (step S704). In this case, 

15 the color temperature of flash light is detected as 
follows. That is, only an image portion/portions 
around the color temperature (e.g., 7000 to 8000K) 
corresponding to flash light of the entire image 
(actually sensed image to be recorded in the recording 

20 medium 200 or 210) undergoes white detection which 
extracts colors included in a white detection range 
based on the black body locus , the color temperatures 
of the detected colors are averaged, and the obtained 
(averaged) color temperature is determined as that of 

25 the flash light. 

Furthermore, the luminance levels of the 
immediately preceding image and actually sensed image 
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are obtained, and the brightness ratio between the 
external light and flash light is calculated based on 
the obtained luminance levels (step S705). The 
brightness can be, e.g., light amounts, light levels 
5 (Y), AE values, or combination thereof. Note that the 
immediately preceding image may have different exposure 
conditions such as an exposure time, aperture value, 
and the like from those of the actually sensed image, 
since an image sensed for the through display is 

10 directly stored. Although the photoelectric conversion 
characteristics of an ideal image sensing device 14 
show direct proportion, the ratio between the luminance 
level and output charge amount varies depending on 
luminance levels in practice. Hence, charge signals 

15 are output at different ratios in correspondence with 
the luminance level obtained based on only the external 
light and that obtained upon sensing an image while 
emitting flash light, and the sensitivity varies. 
Furthermore, when respective pixels are covered by 

20 color filters, the photoelectric conversion 

characteristics also often vary depending on colors. 
For this reason, differences such as an exposure 
condition difference, sensitivity difference, 
sensitivity differences for respective colors (spectral 

25 sensitivity differences), and the like must be taken 
into account . 

In step S706, the external light color 
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temperature and flash light color temperature are 
weighted on the basis of the light amount ratio 
calculated in step S705, and a final color temperature 
is calculated by calculating the average of the sum of 
5 the obtained values. The actually sensed image 

undergoes knovm white balance correction corresponding 
to the calculated color temperature {step S707), thus 
ending the process . 

Fig. 8 is a detailed timing chart from when the 

10 immediately preceding image is captured (step S128) 
until the image sensing process is completed (step 
S129), when an image is sensed while emitting flash 
light in accordance with an image sensing instruction 
issued upon depression of the switch SW2 in step S125 

15 in Fig. 3. 

When an image sensing instruction issued upon 
depression of the switch SW2 is detected (tl), the 
system controller 50 captures an image (immediately 
preceding image) obtained from the image sensing device 

20 14 at that time (corresponding to step S128 in Fig. 3). 
When the image sensing device 14 comprises a CCD and 
has an electronic shutter function, the shutter 12 is 
set in an open state to display, e.g., a through image 
on the image display unit 28, and a charge signal 

25 accumulated by exposing the image sensing device 14 
from the end of a clear period of respective pixels 
until the start of the next clear period is read out. 
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thus obtaining an Image signal. As shown In Fig. 8, 
since the clear period starts In response to a vertical 
synchronization signal VD, the Immediately preceding 
image in this case is obtained by transferring charges 
5 accumulated on respective pixels during a period from 
the previous clear period end time tO to time t2 to a 
transfer CCD in response to read pulses synchronized 
with the vertical synchronization signal VD at time t2. 
Furthermore, the clear period starts at time t2, and 

10 the remaining charges on the respective pixels after 
transfer are cleared. The charges transferred to the 
transfer CCD are sequentially transferred and read out 
in the vertical and horizontal directions by a known 
method until the next image read process starts . 

15 At time t2, the exposure controller 40 opens the 

shutter 12 having the diaphragm function in accordance 
with the aperture value on the basis of the photometry 
data stored in the Internal memory of the system 
controller 50 or the memory 52 (step S321 in Fig. 5). 

20 At time t3, the clear period ends, and charge 

accumulation on the image sensing device 14 starts. 
Upon outputting a read pulse of exposure end at time t4, 
a charge signal begins to be read out from the image 
sensing device 14 , and sensed image data is written in 

25 the memory 30 via the A/D converter 16, image processor 
20, and memory controller 22 or directly from the A/D 
converter 16 via the memory controller 22. Also, that 
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data is stored in the internal memory of the system 
controller 50 or the memory 52 as external light 
photometry data (step S322 in Fig. 5). Likewise, 
during the next exposure period (t5 to t6), the flash 
5 is controlled to emit light in a predetermined amount, 
pre-emission photometry data is also stored in the 
internal memory of the system controller 50 or the 
memory 52, and an actual emission amount is calculated 
from the difference between the two photometry data 

10 (step S323 in Fig. 5). 

After the actual emission amount is determined, 
the exposure controller 40 opens the shutter 12 having 
the diaphragm function in accordance with the aperture 
value on the basis of the external light photometry 

15 data and pre-emission photometry data stored in the 
internal memory of the system controller 50 or the 
memory 52 (step S301 in Fig. 5), so as to set the 
exposure conditions for the actual image sensing 
process . 

20 Upon completion of the setup of the exposure 

conditions , when the clear period of the image sensing 
device 14 ends (t7), charge accumulation starts. At 
this time, the flash is controlled to emit light in 
accordance with the flash emission amount stored in the 

25 internal memory of the system controller 50 or the 

memory 52 (step S304 in Fig. 5). After an elapse of a 
predetermined period of time (t8), the shutter 12 is 
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closed (step S306 in Fig. 5). After the shutter 12 is 
closed, a read pulse for the first field is output (t9) 
to read out a charge signal from the image sensing 
device 14, and sensed image data is then written in the 
5 image data memory 30 via the image sensing device 14, 
A/D converter 16, image processor 20, and memory 
controller 22 or directly from the A/D converter 16 via 
the memory controller 22. Also, that data is stored in 
the internal memory of the system controller 50 or the 

10 memory 52. Likewise, a read pulse for the second field 
is output (tlO), and sensed image data is written in 
the memory 30. As a result, two field images are 
written, i.e., a frame image for all pixels is stored. 
Using the immediately preceding image and the 

15 first and second field images obtained by the 

aforementioned timing control, the processes in steps 
S703 to S706 in Fig. 7 are executed. 

In the above embodiments, the color temperature 
and luminance level of external light are obtained from 

20 the immediately preceding image which is a through 

image (displayed on the EVF) sensed after the shutter 
switch SW2 is turned ON and before an actual image 
sensing starts. Alternatively, the color temperature 
and luminance level of external light may be regularly 

25 obtained from through images after the shutter switch 
SW is turned on. Further, a ratio of brightness 
between the flash light and external light may be 
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obtained by an external light sensor. 

In this way, even when an Image is sensed by 
emitting flash light, color balance correction of the 
sensed image can be appropriately attained. 
5 <Other Embodiment > 

The present invention can be applied to a system 
constituted by a plurality of devices (e.g., image 
processing apparatus such as a host computer, interface, 
camera head) or to an apparatus comprising a single 
10 device (e.g., digital camera, digital video camera). 

Further, the object of the present invention can 
also be achieved by providing a storage medium storing 
program codes for performing the aforesaid processes to 
a computer system or apparatus (e.g., a personal 
15 computer), reading the program codes, by a CPU or MPU 
of the computer system or apparatus, from the storage 
medium, then executing the program. 

In this case, the program codes read from the 
storage medium realize the functions according to the 
20 embodiment, and the storage medium storing the program 
codes constitutes the invention. 

Further, the storage medium, such as a floppy 
disk, a hard disk, an optical disk, a magneto-optical 
disk, CD-ROM, CD-R, a magnetic tape, a non- volatile 
25 type memory card, and ROM, and computer network, such 

as LAN (local area network) and WAN (wide area network), 
can be used for providing the program codes . 
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Furthermore, besides aforesaid functions 
according to the above embodiment are realized by 
executing the program codes which are read by a 
computer, the present invention includes a case where 
5 an OS (operating system) or the like working on the 
computer performs a part or entire processes in 
accordance with designations of the program codes and 
realizes functions according to the above embodiment. 

Furthermore, the present invention also Includes 

10 a case where, after the program codes read from the 

storage medium are written in a function expansion card 
which is inserted into the computer or in a memory 
provided in a function expansion unit which is 
connected to the computer, CPU or the like contained in 

15 the function expansion card or unit performs a part or 
entire process in accordance with designations of the 
program codes and realizes functions of the above 
embodiment . 

In a case where the present invention is applied 
20 to the aforesaid storage mediiim, the storage medium 
stores program codes corresponding to the flowcharts 
shown in Figs. 2 to 7 described in the embodiment. 

The present invention is not limited to the above 
embodiments and various changes and modifications can 
25 be made within the spirit and scope of the present 
invention. Therefore to apprise the public of the 
scope of the present invention, the following claims 
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are made. 
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